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(57) ABSTRACT 

In general, the present invention is directed to systems and 
methods for finding the position and shape of an object using 
video. Tlie invention includes a system with a video camera 
coupled to a computer in which the computer is configured 
to automatically provide object segmentation and 
identification, object motion tracking (for moving objects), 
object position classification, and behavior identification. In 
a preferred embodiment, the present invention may use 
background subtraction for object identification and 
tracking, probabilistic approach with expectation- 
maximization for tracking the motion deteaion and object 
classification, and decision tree classification for behavior 
identification. Thus, the present invention is capable of 
automatically monitoring a video image to identify, track 
and classify the actions of various objects and the object's 
movements within the image. The image may be provided in 
real time or from storage. The invention is particularly 
useful for monitoring and classifying animal behavior for 
testing drugs and genetic mutations, but may be used in any 
of a number of other surveillance applications. 
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SYSTEM AND METHOD FOR OBJECT systems available for monitoring mouse behavior, those that 

IDENTIFICATION AND BEHAVIOR identify individual behaviors and those that identify only the 

CHARACTERIZATION USING VIDEO location of the mouse. 

ANALYSIS The most basic state-of-art mouse behavior analysis sys- 

5 tems rely on traditional analog technologies that can only 

GOVERNMENT RIGHTS NOTICE treat a mouse as an indivisible object and identify the mouse 

„ ^. - . • 1 • .u- -i- location. All the information about a mouse is packed as a 

Portions of the matenal m this specification arose as a ^ ^he space and a lot of important information about 

result of Government support under contracts MH58964 and ^^^^ behavior is lost. The best these systems can do is to 

MH58964-02 between Clever Sys., Inc. and Ihc National fi^d the position of the mouse. Systems like San Diego 

Institute of Mental Health, National Institute of Health, The instruments' Photobeam and AccuScan Instruments Inc.'s 

Government has certam rights m this mvention. Digiscan Line of Animal Activity Monitoring, Columbus, 

BACKGROUND OF THE INVENTION simple and rudimentary photo-beams to detect 

and track the positions of mouse. These systems trackers 

1. Technical Field -15 have a very low spatial resolution, limiting their output to a 
The invention relates generaUy to object identification and ^^S*^ measure of the animal* s activity. They cannot differ- 

recognition. More particularly, one aspect of the invention is entiate even such basic behaviors as locomotion and cir- 

directed to monitoring and characterization of an object in ^^S* Adding a time line for the locus of mouse point is all 

an image, for example an animal or a person, using video they can offer. Other animal location type systems used to 

analysis 20 nionitor animal motion include those described in U.S. Pat. 

2. Background Art ^o^ ^'i^^pL^.^.^;^^-^^^' 

1 . ^ , ^ ^ ... 734; and 5,816,256. 

\^deo analysis has developed over the past few decades to .,^f.«o •« *i, « ♦u * * a 

. • * 1 4 r t. • • r The other systems m the field are the systems that identify 

become an mtegral part of machme operations m manufac- • j- j i u u • tt. • j i • 

^ . . i- ^ 1 -I I- individual behavior usme video. The exjstine video analysis 

turmg using machme automation. For example, video obiect „ / kt u /rr^u a. v 

^ . . * J v.- L i_ J . • . 25 systems (e.g. Noldus Observer/Ethovision, Sterlmg, VA; 

recognition and pattern recognition has been used to orient tt^/o t ti * tti^ a o t . \ , 

,^1. . . f J . r u' ■ J Image, Hampton, UK; AccuScan Instruments Inc. s 

and ahgn vanous pieces of a product for machmmg and videoScan2000 System; and San Diego Instruments Poly- 

assembly m vanous manufacmnng industnes. One such use ™ , . o t>»- a- it\ j * . . 

. . r » • c • J * • . * J Track system, San Diego, Cahf.) do not meet expectations 

IS in the manuiacturmg or senuconductor mtegrated circuits -.u i^- j • L j ^ i 

J - 1 * * I - 1 .I.- either. Digitized images firom video arc used to capture the 

and microelectronic packaging. In this case, pattern recog- u^a f ™ a a j * u . *u 

. , \ . J \ • r 1 30 hody or mouse and provide quantitative data about the 

nition has made great mroads because the size of the work movements of the animal and the pattern of 

product IS microscopic and oaentation and ahgnment of the ^^^^ ^^^^^^^^ ^^^^^ ^^^^^ ^^^^^ 

work product is thus far too tedious for a human bemg to do ^ ^^^^^^ ^ ^^^^^ ^^^^^^ 

consistently and accurately over a large number of pieces. ^ ^^^^^ ^^^^ information is preserved. 

In recent years, miHtary has earned out research to use 3^ However, they can only make use of a few simple features, 

video to track moving targets such as tanks and vehicles, in example, the mass center of the animal (e.g., mouse) is 

the scene. Other positioning instruments such as global calculated and used as a means for tracking the animal (e.g., 

positionmg system will be used to assist such tracking. ^ mouse). As such, a lot of information that is critical to 

Another application for video analysis is monitoring identify the animal's behaviors such as different postures, 

animal activity in laboratory testing for the phanmaceutical positions of portions of the animal's body such as limbs, is 

and biological sciences. One particular area is monitoring lost. These systems can only distinguish basic behaviors 

animal behavior to determine the effects of various new such as locomotion, and cannot automatically identify 

drugs or gene changes on a particular type of animal. One simple animal postures such as eating, rearing, and jumping, 

such animal used in laboratory testing is the mouse. not to mention complex behaviors such as skilled reaching. 

Over the last two decades, major technological advances 45 Such behavior identification requires human intervention 

have enabled scientists to build a rich repository of mouse and input. 

models. Model organisms are an important tool for under- In addition, these systems are often developed for rats that 

standing and dissecting human disease and biological pro- remain relatively stationary in shape as they are in locomo- 

cess. Because mice and humans share many of the same tion. However, other animals such as a mouse frequently 

fundamental biological and behavioral processes, this ani- 50 stretch out, making their center of mass much less stable 

mal is one of the most significant laboratory models for than a rat. As the center of gravity shifts rapidly and 

human disease and studying biological processes in mam- firequently, this falsely adds to measures such as distance 

mals. However, the adequate behavioral characterization traveled, making these systems highly inaccurate for mice, 

(behavioral phenotyping — the impact of a genetic manipu- Further, the systems are devised to study white rats on a dark 

lation on visible characteristics of an organism) of geneti- 55 background and are not accurate for tracking other animals 

cally engineered mice is becoming a serious bottleneck in such as brown or black mice. 

the development of animal models; an exponentially The most advanced systems are those offered by Noldus. 

increasing number of genotypes are created, but the behav- The Noldus Observer system has a video camera, TV 

ioral phenotyping is often at best rudimentary or is aban- monitor, a high end VCR, and a PC system, aU hooked 

doned completely. This is because presently the phenotyping eo together. The camera takes video footage of the mouse in a 

process is largely manual, time consuming, and insensitive cage. This video is recorded on videotape, digitized, input 

to subtle phenotypes. into the PC system, and displayed on the computer monitor. 

Mdeo technologies for mouse behavior analysis have Although the human observer can control the recorded video 

been introduced and several products are commercially that is displayed, the human observer still needs to look at 

available. However, these technologies are still primitive 65 the animal on the screen, decide which behavior the animal 

and the functionality of the products is far from adequate for is engaged in, and enter (by typing) the inforaaation into a 

the research purposes. There are presently two types of mechanism provided by the system for storage and later 
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analysis. While this system facilitates observation of of video images. Next, the changes in the shapes, locations, 

behavior, it does not automate it, and is thus prone to human and/or postures of the object of interest may be identified, 

error and extremely labor intensive. The tasks of coding their features extracted, and classified into meaningful 

behavior throughout the day and building a profile of categories, for example, vertical positioned side view, hori- 

behavior for different types of animals and different strains 5 zontal positioned side view, vertical positioned front view, 

of the same animal (e.g., different strains of mouse) is horizontal positioned front view, moving left to right, etc. 

prohibitively time consuming with this equipment. Then, the shape, location, and posture categories may be 
SUMMARY OF THE INTVENTION characterize the object's activity into one of a 

, , • • J. i . number of pre-defined behaviors. For example, if the object 

In general, the present mvention is directed to systems and . . , luu- 

methodsforfindiigpattemsofbehaviorsand/oractivitiesof ^ an anjmal, some pre-defiiied normal behaviors may 

an object using video. The invention includes a system with ^"^1^^^ ft^'J^ ^^^"^6' ^^^.g' ^f^- 

a video camera connected to a computer in which the and pre-defined abnormal behavior may include spmmng 

computer is configured to automatically provide object ^^^^^^^^^ jumpmg in the same spot, etc. The pre-defined 

identification, object motion tracking (for moving objects), behaviors may be stored m a database in the data memory, 

object shape and posture classification, and behavior iden- is The behavior may be characterized using, for example, 

tification. Thus, the present invention is capable of auto- approaches such as rule-based label analysis, token parsing 

matically monitoring a video image to identify, track and procedure, and/or Hidden Markov Modeling (HMM). 

classify the actions of various objects and their movements. Further, the system may be constructed to characterize the 

The video image may be provided in real time from a camera object behavior as new behavior and particular temporal 

and/or from a storage location. The invention is particularly 20 rhythm. 

useful for monitoring and classifying animal behavior for in another preferred embodiment directed toward video 

testing drugs and genetic mutations, but may be used in any analysis of animated objects such as animals, the system 

of a number of surveillance or other applications. operates as foUows. As a preliminary matter, normal pos- 

tn one embodiment the invention includes a system in tures and behaviors of the animals are defined and may be 

which an analog video camera and a video record/playback 25 entered into a Normal Postures and Behaviors database. In 

device (e.g., VCR) are coupled to a video digitization/ analyzing in a first instant, incoming video images are 

compression unit. The video camera may provide a video received. The system determines if the video images are in 

image containing an object to be identified. The video analog or digital fonmat and input into a computer. If the 

digitization/compression unit is coupled to a computer that video images are in analog format they are digitized and may 

is configured to automatically monitor the video image to 30 be compressed, using, for example, an MPEG digitizer/ 

identify, track and classify the actions of the object and its compression unit. Otherwise, the digital video image may be 

movements over time within a sequence of video session input directly to the computer. Next, a background may be 

image frames. The digitization/compression unit may con- generated or updated from the digital video images and 

vert analog video and audio into, for example, MPEG or foreground objects detected. Next, the foreground objects 

other formats. The computer may be, for example, a per- 35 features are extracted. Then, the foreground object shape is 

sonal computer, using either a Windows platform or a Unix classified into various categories, for example, standing, 

platform, or a Macintosh computer and compatible platform. silting, etc. Next, the foreground objea posture is compared 

The computer is loaded and configured with custom soft- to the various predefined postures stored in the database, and 

ware programs (or equipped with firmware) using, for then identified as a particular posture or a new (unidentified) 

example, MATLAB or C/C++ programming language, so as 40 posture. Then, various groups of postures are concatenated 

to analyze the digitized video for object identification and into a series to make up a foreground object behavior and 

segmentation, tracking, and/or behavior/activity character- then compared against the sequence of postures, stored in for 

ization. This software may be stored in, for example, a example a database in memory, that make up known normal 

program memory, which may include ROM, RAM, CD or abnormal behaviors of the animal. The abnormal behav- 

ROM and/or a hard drive, etc. In one variation of the 45 iors are then identified in terms of known abnormal 

invention the software (or firmware) includes a unique behavior, new behavior and/or daily rhythm, 
background subtraction method which is more simple, in one variation of the invention, object detection is 

efficient, and accurate than those previously known. perfonned through a unique method of background subtrac- 

In operation, the system receives incoming video images tion. First, the incoming digital video signal is split into 

from either the video camera in real time or pre-recorded 50 individual images (frames) in real-time. Then, the system 

from the video record/playback unit. If the video is in analog determines if the background image derived from prior 

format, then the information is converted from analog to incoming video needs to be updated due to changes in the 

digital format and may be compressed by the video background image or a background image needs to be 

digitization/compression unit. The digital video images are developed because there was no background image was 

then provided to the computer where various processes are 55 previously developed. If the background image needs to be 

undertaken to identify and segment a predetermined object generated, then a number of frames of video image, for 

from the image. In a preferred embodiment the object is an example 20, will be grouped into a sample of images. Then, 

object (e.g., a mouse) in motion with some movement from the system creates a standard deviation map of the sample of 

frame to frame in the video, and is in the foreground of the images. Next, the process removes a bounding box area in 

video images. In any case, the digital images may be 60 each frame or image where the variation within the group of 

processed to identify and segregate a desired images is above a predetermined threshold (i.e., where the 

(predetenmined) object from the various frames of incoming object of interest or moving objects are located). Then, the 

video. This process may be achieved using, for example, various images within the sample less the bounding box area 

background subtraction, mixture modeling, robust are averaged. Final background is obtained by averaging 

estimation, and/or other processes. 65 5-10 samples. This completes the background generation 

The shape and location of the desired object is then process. However, often the background image does not 

tracked from one frame or scene to another frame or scene remain constant for a great length of time due to various 
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reasons. Thus, the background needs to be recalculated software and hardware system according to the present 

periodically as above or it can be recalculated by keeping invention. An automated method for analysis of mouse 

track of the difference image and note any sudden changes. behavior from digitized 24 hour video has been achieved 

The newly generated background image is next subtracted using the present invention and its digital video analysis 

from the current video unage(s) to obtam foreground areas 5 method for object identification and segmentation, tracking, 

that may include the object of interest. classification. Several different methods and their 

Next, the object identificaUon/detection process is per- algorithms, including Background Subtraction, Probabilistic 
formed. First, regions of mterest (ROI) are obtained by ^ with Expectation-Maximization, and Robust Esti- 
identifying areas where the mtensity difference generated ^.^^^ parameter values by best fitting a set of data 
from the subtraction IS greater than a predetermmed n^easurements and results proved successfiil. The entire 
threshold, which consti ute potential foreground object(s) behavioral repertoire of indiWdual mice in their home cage 
being sought. Classification of these foreground regior^ of categorized using successive iterations by manual vid- 
interest will be performed usir^ the sizes of the ROls .^^ape analysis. These manually defined behavior categories 
distances among these ROIs, threshold of intensity, and constituted the basis of automatic classification. Classifica- 
connectedness, to thereby identify the foreground objects. ^-^^ ^^^^^a (based on features extracted from the fore- 
Next the foreground objea Identification/detection process ^ object such as shape, position, movement) were 
may be refilled by adaptively leammg histograms of fore- derived and fitted into a decision tree (DT) classification 
^r^f A Tu^ edge detection to more accurately algorithm. The decision tree could classify ahnost 500 
identffy the desired objcct(s)^Fmally, the information iden- ^ ^^^^^^^^ -^^^ 5 ^^^^^^^ ^^^^^^ ^^^^^^ ^-^^ 
tifymg the desired foreground object is output. TTie process accuracy over 93%. A simple HMM system has been built 
may then continue with the tracking and/or behavior char- ^^^^ jy^^^j^ programming and has been used to classify 
acterization step(s). classified postures identified by the DT and yields an 

The previous embodiments are particularly appUcable to ahnost perfect mapping from input posture to output behav- 

the study and analysis of mice used in genetic and drug jors in mouse behavior sequences. 

experimentation. One variation of the present invention is 9 c „ - j**^ l ill ^ 

J. , . 1 I . . 11 J • • . '^^ The mvention may identify some abnormal behavior by 

directed particularly to automatically determining the • • c r 1 T 1 ■ 

w u ' t u 4-4' c u ^ using video image information (for example, stored m 

behavioral characteristics of a mouse in a home cage. The \ ^ t t. 1 - 1 . J. u ^ 

J * 4- c 1 t- 4 r memory) 01 known abnormal animals to build a video 

need for sensitive detection of novel phenotypes of geneti- / ^i, * u u • x:- t -j • r 

n - I 4 J J J • • 4 J • J J . pro tile tor that behavior. For example, video image of 

cally manipulated or drug-admmiste red mice demands auto- r,„.tv..i • * u i u • c .u 4 4 j 

/ 1 T> u • 1 L 4 r4 u . vertical spinning while hangmg from the cage top was stored 

matron of analyses. Behavioral phenotypes are often best 30 „^ « a a * * n j T*r t. 

4, ^ J , ' . 4 • J u • 4 to memory and used to automatically identify such activity 

detected when mice are unconstramed by experimenter • ^ r- u i-.- t 1. r 

r™ 4 *■ c i 1° rmce. Further, abnormalities may also result from an 

manipulation. Thus, automation of analysis of behavior m a • . 1 * r . 1. 1. ■ 

, ^ . \ c 4 . , . , mcrease in any particular type of normal behavior. Detection 

known environment, for example a home cage, would be a p , 1 u if - i_ u - „ ; . 

A 1 4 1 r J 4 4- u * if- r 01 such uew abnormal behaviors may be achieved by the 

powerful tool for detecting phenotypes resulting from eene , • 1 . r j ^^^^^^^ ^^^^ 

. , J J • • 4 4- A . 4- iT 1 present mvention detecting, for example, segments of 

manipulations or drug administrations. Automation of analy- « uu • *u * ^ * 4U 4 ^ j ^ uio v^l 

ij 11 4-c 4- c 11 1_ u • 4i_ behavior that do not fit the standard profile. The standard 

SIS would allow quantification of all behaviors as they vary «,„fii^ , u« j 1 j 4- 1 . • r 

, 1 f 4- V Ti J r 4 profile may be developed for a particular strain of mouse 

across the daily cycle of activity. Because gene defects . j 4 4' c L ^ r 11. 

J 1 * 1 J- J - I- 11 t • whereas detection of abnormal amounts of a normal be hav- 
causing developmental disorders m humans usually result m k a^* * a u • » 4U . *• 4- 1 
u • 41. \i 1 u .4U f u u • I • f lor can be detected by comparison to the statistical proper- 
changes m the daily rhythm of behavior, analysis of orea- f*u4-j_j <:i 

. f f u u • 4t. J 11- ties of the standard profile, 

nized patterns of behavior across the day may also be 40 _ 

effective in detecting phenotypes in transgenic and targeted automated analysis of the present invention may 

mutant mice. The automated system may also be able to ^^^^ ^^^^ P^^^^^^ behaviors, their amount, 

detect behaviors that do not normally occur and present the duration, and daily cycle for each animal, for example each 

investigator with video clips of such behavior without the commonly used strain of mice. A plurality of such profiles 

investigator having to view an entire day or long period of 45 example, a database in a data memory 

mouse activity to manually identify the desired behavior. computer. One or more of these profile may then be 

The systematically developed definition of mouse behav- ^^^f red to a mouse in question and difference from the 

ior that is detectable by the automated analysis according to P'"*^^^ expressed quantitatively. 

the present invention makes precise and quantitative analy- ^e techniques developed with the present invention for 

sis of the entire mouse behavior repertoire possible for the 50 automation of the categorization and quantification of all 

first time. The various computer algorithms included in the home-cage mouse behaviors throughout the daily cycle is a 

invention for automating behavior analysis based on the powcrfiil tool for detecting phenotypic effects of gene 

behavior definitions ensure accurate and efScient identifica- manipulations in mice. As previously discussed, this tech- 

tion of mouse behaviors. In addition, the digital video oology is extendable to other behavior studies of animals 

analysis techniques of the present invention improves analy- 55 humans, as well as surveiUance purposes. As will be 

sis of behavior by leading to: (1) decreased variance due to described in detail below, the present invention provides 

non-disturbed observation of the animal; (2) increased autonaated systems and methods for automated accurate 

experiment sensitivity due to the greater number of behav- identification, tracking and behavior categorization of an 

iors sampled over a much longer time span than ever before object whose image is captured with video, 

possible; and (3) the potential to be appUed to all common 60 ^RIEF DESCRIPTION OF THE DRAWINGS 

normative behavior patterns, capability to assess subtle ^ix^^ttiuvjo 

behavioral states, and detection of changes of behavior FIG. 1 is a block diagram of one exemplary system 

patterns in addition to individual behaviors. configurable to find the position, shape, and behavioral 

Development activities have been completed to validate characteristics of an object using automated video analysis, 

various scientific definitions of mouse behaviois and to 65 according to one embodiment of the present invention, 

create novel digital video processing algorithms for mouse FIG. 2 is a block diagram of various functional portions 

tracking and behavior recognition, which are embodied in a of a computer system, such as the computer system shown 
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in FIG. 1, when configured to find the position, shape, and FIG. 10 is a chart illustrating one example of various 

behavioral characteristics of an object using automated mouse state transitions used in characterizing mouse behav- 

video analysis, according to one embodiment of the present ior including: Horizontal (HS); Cuddled up (CU); Partially 

invention. reared (PR); Vertically Reared (VR); and Forward Back 

FIG. 3 is a flow chart of a method of automatic video 5 (FB), along with an indication of duration of these states 

analysis for object identification and characterization, based on a sample, according to one variation of the present 

according to one embodiment of the present invention. invention as apphed for monitoring and characterizing 

HG. 4 is a flow chart of a method of automatic video ^^^^ behavior, 

analysis for object identification and characterization, DESCRIPTION OF THE PREFERRED 

according to another embodiment of the present invention, EMBODIMENTS 

FIG. 5 is a flow chart of a method of automatic video The past few years have seen an increase in the integration 

analysis for object detection and identification, accordmg to ^^eo camera and computer technologies. Today, the 

one variation of the present invention, integration of the two technologies aUows video images to 

FIG. 6 illustrates a sample video image frame with a be digitized, stored, and viewed on small inexpensive 

mouse in a rearing up posture as determined using one computers, for example, a personal computer. Further, the 

variation of the present invention to monitor and character- processing and storage capabilities of these small inexpen- 

ize mouse behavior. sive computers has expanded rapidly and reduced the cost 

FIG. 7 A is a first video image frame in a sequence with for performing data and computational intensive applica- 
a mouse in an eating posture for illustrating background 20 lions. Thus, video analysts systems may now be configured 

generation for a background subtraction process according to provide robust surveillance systems that can provide 

to one variation of the present invention as applied for automated analysis and identification of various objects and 

monitoring and characterizing mouse behavior. characterization of their behavior. The present invention 

FIG. 7B is a copy of the first video image frame of FIG. provides such systems and related methods. 
7A in which the process has extracted an area of the video 25 In general, the present invention can automatically find 

image related to the mouse in the foreground resulting in a the patterns of behaviors and/or activities of a predetermined 

"hole" which will be filled up when other frames are object being monitored using video. The invention includes 

averaged with it for a background subtraction process a system with a video camera connected to a computer in 

according to one variation of the present invention as which the computer is configured to automatically provide 
applied for monitoring and characterizing mouse behavior. 30 object identification, object motion tracking (for moving 

FIG. 7C is the residting background image for a video clip objects), object shape and posture classification, and behav- 

including the first video image frame of FIG. 7 A converted ior identification. In a preferred embodiment the system 

as shown in FIG. 7B and averaged with subsequent video includes various video analysis algorithms. The computer 

images, according to one variation of the present invention processes analyze digitized video with the various algo- 
as applied for monitoring and characterizing mouse behav- 35 rithms so as to automatically monitor a video image to 

ior. identify, track and classify the actions of one or more 

FIG. 8A is a difference image between foreground and predetermined objects and its movements captured by the 

background for the image shown in FIG. 7A, according to ^^^^ as it occurs from one video frame or scene to 

one variation of the present invention as applied for moni- another. The system may characterize behavior by accessing 

taring and characterizing mouse behavior. ^ database of object information of known behavior of the 

FIG. 8B is the image shown in BG. 7A after completing Predetermined object. The image to be analyzed may be 

a threshold process for identifying the foreground image of P'^^^^'^ ''"^^ ^^"^ '"^^^ 

the mouse which is shown as correctly identified, according storage. 

to one variation of the present invention as appUed for various exemplary embodiments described in detail as 

monitoring and characterizing mouse behavior. follows, the invention is configured to enable monitoring 

FIG. 8C is a video image frame showing the foreground classifying of animal behavior that results from testing 

mouse object correctly identified by the system as identified ^7^' ^7,"^^" mutations on ammals. However as indi- 

with a polygon outUne, according to one variation of the "'^"^ '^^^^ the system may be smiilarly configured for use 

present invention as applied for monitoring and character- , f ^Z^^!"*'"' f surveiUance or other apphcations. For 

izing mouse behavior example, the invention can be applied to various situations 

™^ n . . . , '. _ ^ . . in which tracking moving objects is needed. One such 

FIG. 9A is a video image frame showmg a mouse eating situation is security surveillance in pubUc areas like airports, 

to demonstrate a b-sphne approach to object location and ^ases, or home security systems. The system may be 

outhne Identification accordmg to one variation of the ^^^^ automatically identifying and notifying proper law 

present mvention as apphed for monitormg and character- enforcement officials if a crime is being commifted and/or a 

izing mouse behavior. particular behavior being monitored is identified. The sys- 

HG. 9B is a computer generated image showing the tem may be useful for monitoring of parking security or 

outline of the foreground mouse shown in FIG. 9A after moving traffic at intersections so as to automatically identify 

edge segmentation, according to one variation of the present and track vehicle activity. The system may be configured to 
invention as appUed for monitoring and characterizing go automatically determine if a vehicle is speeding or has 

mouse behavior. performed some other traffic violation. Further, the system 

FIG. 9C is a computer generated image of the outline of may be configured to automatically identify and characterize 

the foreground mouse shown in FIG. 9A as derived from the human behavior involving guns or human activity related to 

outline of the mouse shown in FIG, 9B as generated from a robberies or thefts. Similarly, the invention may be capable 
b-sphne process, according to one variation of the present 65 of identifying and understanding subtle behaviors involving 

invention as applied for monitoring and characterizing portions of body such as forehmb and can be apphed to 

mouse behavior. identify and understand human gesture recognition. This 
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could help deaf individuals communicate. The invention 
may also be the basis for computer understanding of human 
gesture to eahance the present human-computer interface 
experience, where gestures will be used to interface with 
computers. The economic potential of applications in 
computer-human interface applications and in surveillance 
and monitoring applications is enormous. 

In one preferred embodiment illustrated in FIG. 1, the 
invention includes a system in which an analog video 
camera 105 and a video storage/retrieval unit 110 may be 
coupled to each other and to a video digitization/ 
compression unit 115. The video camera 105 may provide a 
real time video image containing an object to be identified. 
The video storage/retrieval unit 110 may be, for example, a 
VCR, DVD, CD or bard disk unit. The video digitization/ 
compression unit 115 is coupled to a computer 150 that is 
configured to automatically monitor a video image to 
identify, track and classify the actions (or state) of the object 
and its movements (or stillness) over time within a sequence 
of images. The digitization/compression unit 115 may con- 
vert analog video and audio into, for example, MPEG 
format, Real Player format, etc. The computer may be, for 
example, a personal computer, using either a Windows 
platform or a Unix platform, or a Macintosh computer and 
compatible platform. In one variation the computer may 
include a number of components such as (1) a data memory 
151, for example, a hard drive or other type of volatile or 
non-volatile memory; (2) a program memory 152, for 
example, RAM, ROM, EEPROM, etc. that may be volatile 
or non-volatile memory; (3) a processor 153, for example, a 
microprocessor; and (4) a second processor to manage the 
computation intensive features of the system, for example, a 
math coprocessor 154. The computer may also include a 
video processor such as an MPEG encoder/decoder. 
Although the computer 150 has been shown in FIG. 1 to 
include two memories (data memory 151 and program 
memory 152) and two processors (processor 153 and math 
co-processor 154), in one variation the computer may 
include only a single processor and single memory device or 
more then two processors and more than two memory 
devices. Further, the computer 150 may be equipped with 
user interface components such as a keyboard 155, elec- 
tronic mouse 156, and display unit 157. 

In one variation, the system may be simplified by using all 
digital components such as a digital video camera and a 
digital video storage/retrieval unit 110, which may be one 
integral unit. In this case, the video digitization/compression 
unit 115 may not be needed. 

The computer is loaded and configured with custom 
software program(s) (or equipped with firmware) using, for 
example, MATLAB or C/C++ programming language, so as 
to analyze the digitized video for object identification and 
segmentation, tracking, and/or behavior/activity character- 
ization. This software may be stored in, for example, a 
program memory 152 or data memory that may include 
ROM, RAM, CD ROM and/or a hard drive, etc. In one 
variation of the invention the software (or firmware) 
includes a unique background subtraction method which is 
more simple, cflBcient, and accurate than those previously 
known which will be discussed in detail below. In any case, 
the algorithms may be implemented in software and may be 
understood as unique functional modules as shown in FIG. 
2 and now described. 

Referring to FIG. 2, the system is preloaded with standard 
object information before analyzing an incoming video 
including a predetermined object, for example, a mouse. 
First, a stream of digital video including a known object with 
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known characteristics may be fed into the system to a 
standard object classifier module 220. A user may then view 
the standard object on a screen and identify and classify 
various behaviors of the standard object, for example, 

5 standing, sitting, lying, normal, abnormal, etc. Data infor- 
mation representing such standard behavior may then be 
stored in the standard object behavior storage modules 225, 
for example a database in data memory 151. Of course, 
standard object behavior information data sets may be 

jQ loaded directly into the standard object behavior storage 
module 225 from another system or source as long as the 
data is compatible with the present invention protocols and 
data structure. In any case, once the standard object behavior 
data is entered into the standard object behavior storage 

J 5 module 225, the system may be used to analyze and classify 
the behavior of one or more predetermined objects, for 
example, a mouse. 

In the automatic video analysis mode, digital video (either 
real-time and/or stored) of monitored objects to be identified 

20 and characterized is input to an objea identification and 
segregation module 205. This module identifies and segre- 
gates a predetermined type of object from the digital video 
image and inputs it to an object tracking module 210. The 
object tracking module 210 facilitates tracking of the pre- 

25 determined object from one frame or scene to another as 
feature information. This feature information is then 
extracted and input to the object shape and posture classifier 
215. This module classifies the various observed states of the 
predetermined object of interest into various shape and 

30 posture categories and sends it to the behavior identification 
module 230. The behavior identification module 230 com- 
pares the object shape, motion, and posture information with 
shape, motion, and posture information for a standard object 
and classifies the behavior accordingly into the predefined 

35 categories exhibited by the standard object, including 
whether the behavior is normal, abnormal, new, etc. This 
information is output to the user as characterized behavior 
information on, for example, a display unit 157. 

Referring now to FIG. 3, a general method of operation 

40 for one embodiment of the invention will be described. In 
operation, in the video analysis mode the system may 
receive incoming video images at step 305, from the video 
camera 105 in real time, pre-recorded from the video 
storage/retrieval unit 110, and/or a memory integral to the 

45 computer 150. If the video is in analog format, then the 
information is converted from analog to digital format and 
may be compressed by the video digitization/compression 
unit 115. The digital video images are then provided to the 
computer 150 for various computational intensive process- 
so ing to identify and segment a predetermined object from the 
image. In a preferred embodiment, the object to be identified 
and whose activities is to be characterized is a moving 
object, for example a mouse, which has some movement 
from frame to frame or scene to scene in the video images 

55 and is generally in the foreground of the video images. In 
any case, at step 310 the digital images may be processed to 
identify and segregate a desired (predetermined) object from 
the various frames of incoming video. This process may be 
achieved using, for example, background subtraction, mix- 

60 ture modeling, robust estimation, and/or other processes. 
Next, at step 315, various movements (or still shapes) of 
the desired object may then be tracked from one frame or 
scene to another frame or scene of video images. As will be 
discussed in more detail below, this tracking may be 

65 achieved by, for example, tracking the outline contour of the 
object from one frame or scene to another as it varies from 
shape to shape and/or location to location. Next, at step 320, 
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the changes in the motion of the object, such as the shapes, 
locations, and postures of the object of interest, may be 
identified and their features extracted and classified into 
meaningful categories. These categories may include, for 
example, vertical positioned side view, horizontal positioned 
side view, vertical positioned front view, horizontal posi- 
tioned front view, moving left to right, etc. Then, at step 325, 
the states of the object, for example the shape, location, and 
posture categories, may be used to characterize the objects 
activity into one of a number of pre-defined behaviors. For 
example, if the object is an animal, some pre-defined normal 
behaviors may include sleeping, eating, drinking, walking, 
running, etc., and pre-defined abnormal behavior may 
include spinning vertical, jumping in the same spot, etc. The 
pre-defined behaviors may be stored in a database in the data 
memory 151. 

Types of behavior may also be characterized using, for 
example, approaches such as rule-based label analysis, token 
parsing procedure, and/or Hidden Markov Modeling 
(HMM). The HMM is particularly helpful in characterizing 
behavior that is determined with temporal relationships of 
the various motion of the object across a selection of frames. 
From these methods, the system may be capable of charac- 
terizing the object behavior as new behavior and particular 
temporal rhythm. 

Referring now to FIG. 4 a more detailed description of 
another preferred embodiment will be described. In this case 
the system is directed toward video analysis of animated 
objects such as animals. As a preliminary matter, at step 415 
video of the activities of a standard object and known 
behavior characteristics are input into the system. This 
information may be provided from a video storage/retrieval 
unit 110 in digitized video form into a standard object 
classified module 220. This information may then be manu- 
ally categorized at step 416 to define normal and abnoranal 
activities or behaviors by a user viewing the video images on 
the display imit 157 and inputting their classifications. For 
example, experts in the field may sit together watching 
recorded scenes. They may then define, for example, an 
animal's (e.g., a mouse) bchavior(s), both qualitatively and 
quantitatively, with or without some help from systems like 
the Noldus Observer system. These cataloged behaviors 
may constitute the important posture and behavior database 
and are entered into a storage, for example a memory, of 
known activity of the standard object at step 420. This 
information provides a point of reference for video analysis 
to characterize the behavior of non-standard objects whose 
behaviors/activities need to be characterized such as geneti- 
cally altered or drug administered mice. For example, nor- 
mal postures and behaviors of the animals are defined and 
may be entered into a normal postures and behaviors data- 
base. 

Once information related to characterizing a standard 
object (s) is established, the system may then be used to 
analyze incoming video images that may contain an object 
for which automated behavior characterization is desired. 
First, at step 405, incoming video images are received. Next, 
at decision step 406, the system determines if the video 
images are in analog or digital format. If the video images 
are in analog format they are then digitized at step 407. The 
video may be digitized and may be compressed, using, for 
example, a digitizer/compression unit 115 into a convenient 
digital video format such as MPEG, RealPlayer, etc. 
Otherwise, the digital video image may be input directly to 
the computer 150. Now the object of interest is identified 
within the video images and segregated for analysis. As 
such, at step 408, a background may be generated or updated 
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from the digital video images and foreground objects includ- 
ing a predetermined object for behavior characterization 
may be detected. For example, a mouse in a cage is detected 
in the foreground and segregated from the background. 

5 Then, at step 409, features such as centroid, the principal 
orientation angle of the object, the area (number of pixels), 
the eccentricity (roundness), and the aspect ratio of the 
object, and/or shape in terms of convex hull or b-spline, of 
the foreground object of interest (e.g., a mouse) are 

30 extracted. Next, at step 410, the foreground object shape and 
postures are classified into various categories, for example, 
standing, sitting, etc. 

Then, at step 411, the foreground object (e.g., a mouse) 
posture may be compare to the various predefined postures 

35 in the set of known postures in the standard object storage 
of step 420, which may be included in a database. At steps 
412, the observed postures of the object contained in the 
analyzed video image may be classified and identified as a 
particular posture known for the standard object or a new 

20 previously unidentified posture. Next, at step 413, various 
groups of postures may be concatenated into a series to make 
up a foreground object behavior that is then compared 
against the sequence of postures, stored in for example a 
database in memory, that make up a known standard object 

25 behavior. This known standard behavior is, in a preferred 
embodiment, normal behavior for the type of animal being 
studied. However, the known activity of the standard object 
may be normal or abnormal behavior of the animal. In either 
case, at step 414, the abnormal behaviors are then identified 

30 in terms of (1) known abnormal behavior; (2) new behavior 
likely to be abnormal; and/or (3) daily rhythm differences 
likely to be abnormal behavior. Known normal behavior 
may also be output as desired by the user. This information 
is automatically identified to the user for their review and 

35 disposition. In one variation of the invention, the informa- 
tion output may include behavior information that is com- 
patible with current statistical packages such as Systat and 
SPSS. 

In one embodiment of the invention as illustrated in FIG. 

40 5, object detection is performed through a unique method of 
background subtraction. First, at step 405, incoming video is 
provided to the system for analysis. This video may be 
provided by digital equipment and input to the object 
identification and segregation module 205 of the computer 

45 150. Next, at step 505, the incoming digital video signal may 
be split into individual images (frames) in real-time. This 
step may be included if it is desired to carry out real-time 
analysis. Then, at decision step 506, the system determines 
if the background image needs to be developed because 

50 there was no background image developed previously or the 
background image has changed. If the background image 
needs to be generated or updated, then at step 507 a 
background image is generated by first grouping a number 
of frames or images into a sample of video images, for 

55 example 20 frames or images. The background may need to 
be updated periodically due to changes caused by, for 
example, lighting and displacement of moveable objects in 
the cage, such as the bedding. Then, at step 508 the system 
generates a standard deviation map of the group of images. 

60 Next, at step 509, an object(s) bounding box area is identi- 
fied and removed from each frame or image to create a 
modified frame or image. The bounding box area is deter- 
mined by sensing the area wherein the variation of a feature 
such as the standard deviation of intensity is above a 

65 predetermined threshold. Thus, an area in the digitized video 
image where the object of interest in motion is located is 
removed leaving only a partial image. Then, at step 510, the 
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various modified images within the group, less the bounding accurately identify the desired object. An edge detection 

box area, are combined, for example averaged, to create a mechanism such as Prewitt operator is apphed to the original 

background image at step 511. image. Adaptive thresholds for edge detections can be used. 

Since varying pixels are not used in averaging, "holes" 9'"=^ ^ obtained, the actual boundary of the 

will be created in each image that is being used in the 5 foreground object K assumed to be made up of one or more 

averaging process. Over time, not aU frames will have these f events m the edge map, i.e. the actual contour of the 

holes at the same location and hence, a complete background ff'g"''^'^ objects comprises edges m the edge map. ITie 

image is obtained after the averaging pro^ss. Final back- ^^^""^ ^^^J^'"'^ foreground object >s broken 

ground is obtained by averaging 5-10 samples. THis com- f^f 1^"",?'.? ^'=2™°'" "^^'^ 

pletes at least one iteration of the background generation lo map bat are closest to these contoi^ segments accordmg to 

process e- & a distance metnc are found to be the desired contour. One 

„ ' , . . , , . , exemplary distance metric is the sum of absolute normal 

The background unage does not remain constant for a distance to the edge map segment from each point in the 
great length of Ume due to vanous reasons. For example the closed contour of the "detected" foreground object. Finally, 
beddmg ma mouse cage can shift due to the activity of the a, siS the information identifying the desired fore- 
mouse. External factors such as change m illumination groundobjectisoutput.Theprocessmay then continue with 
conditions also require background image recalculations. If ticking and/or behavior characterization steps, 
the camera moves, then, background might need to be Tn,„ „„„• „™u^j' ♦ n i- l, 
changed. TTius, the background t^icaUy needs to be recal- He^ffZ ir^. Jna T h T ^'°"?h '' ''PP'?"'''%'° 
culattj periodicaUy as described above or it can be recal- ' 'ri^&V.f f .^^ f ^ °^ ' 

1, 1 • . 1 f *u j-rc • J . 20 particular object of interest present in a Video image, e.g., an 

culated by keeping track of the difference image and note „ u,. u- t * it • . " 

AA u u • *u u r animal, a human, a vehicle, etc. However, the mvention is 

any sudden changes such as an increase in the number of . i- ui . *u *a ^ , . ^ 

, , \ • 1 • .u A'cc ' particularly applicable to the study and analysis of 

particular color (e.g., white) pixels in the difference unage or o^.r^oie f«,#^o4- ^ a/ I - 

fu r 1 I. r.u 1 t r t . > auimals used lor testing ucw drugs End/or genetic mutations. 

the appearance ofpatches of the particular color (e.g., white) a«. « u f - * ■ r *u • x- 

• 1 • f *u j-ff • T ^ such, a number of vanations of the mvention related to 

pixels in another area of the difference image. In any case, • uu • r • ti l j •. i 

f. t * J L 1 J • u 25 determmmg changes in behavior of mice will be described 

the newly generated background image may then be com- - „„„ f.^-. if, . , c • 

u- A -lu • u 1 J ' : . ^ a&i2Ln below using examples of video images 

bined with any existing background image to create a new obtained & ^ e^" 

background image at step 511. ^ . . ^ , . . 

™ 1 *jt-i J- • ... variation or the present invention is designed par- 

The newly generated background image is next, at step ^icularly for the purpose of automatically detern^ing the 

512, subtracted from the current video image(s) to obtam 3^ behavioral character^ics of a mouse. THe need for sensitive 

foreground areas that may include the object of mterest. detection of novel phenotypes of genetically manipulated or 

Further, If the background does not need to be updated as drug-administered mice demands automaton of analyses, 

determmed at decision step 506. then the process may ^^^^^.^^^^ phenotypes are often best detected when mice 

proceed to step 512 and the back^uiid miage is subtracted unconstrained by experimenter manipulation. Thus. 

from the current miage, leaving the foreground objects. 3^ automation of analysis of behavior in a home cage would be 

Next, at steps 513-18, the object identification/detection a preferred means of detecting phenotypes resulting from 

process is performed. First, at step 513, regions of interest gene manipulations or drug admintstralions. Automation of 

(ROI) are obtamed by identifying an area where the intensity analysis as provided by the present invention will allow 

difference is greater than a predetermined threshold, which quantification of all behaviors and may provide analysis of 

constitute potential foreground objecl(s) bemg sought. Clas- 40 the mouse's behavior as they vary across the daily cycle of 

sification of these foreground regions of interest wiU be activity. Because gene defects causing developmental dis- 

performed using the sizes of the ROIs, distances among orders in humans usually result in changes in the daily 

these ROIs, threshold of intensity, and connectedness to rhythm of behavior, analysis of organized patterns of behav- 

identify the foreground objects. Next, the foreground object ior across the day may be effective in detecting phenotypes 

identification/detection process may be refined by utiUzing 45 in transgenic and targeted mutant mice. The automated 

mformauon about the actual distribution (histograms) of the system of the present invention may also detect behaviors 

mtensity levels of the foreground object and using edge that do not normally occur and present the investigator with 

detection to more accurately identify the desired object(s). video clips of such behavior without the investigator having 

At step 514, during both the background generation and to view an entire day or long period of mouse activity to 

background subtraction steps for object identification, the 50 manually identify the desired behavior, 

system continuously maintains a distribution of the fore- The systematically developed definition of mouse behav- 

ground object mtensities as obtained. Alower threshold may ior that Ls detectable by the automated analysis of the present 

be used to thereby permit a larger amount of noise to appear invention makes precise and quantitative analysis of the 

m the foreground image in the form of ROIs. Thus, at step entire mouse behavior repertoire possible for the first time. 

514, a histogram is then updated with the pixels in the ROI. 55 The various computer algorithms included in the invention 

At step 515, plotting a histogram of all the intensities of a for automating behavior analysis based on the behavior 

particular color pixels over many images, provides a definitions ensure accurate and efficient identification of 

bi-modal shape with the larger peak corresponding to the niouse behaviors. In addition, the digital video analysis 

foreground object's intensity range and the smaller peak techniques of the present invention improves analysis of 

corresponding to the noise pixels in the ROrs images. Now, go behavior by leading to: (1) decreased variance due to 

at step 516, having "learned" the intensity range of the non-disturbed observation of the animal; (2) increased 

foreground object, only those pixels in the foreground object experiment sensitivity due to the greater number of behav- 

that conform to this intensity range are selected, thereby iors sampled over a much longer time span than ever before 

identifying the foreground object more clearly even with possible; and (3) the potential to be appUed to all common 

background that is fairly simUar. ^5 normative behavior patterns, capability to assess subtle 

In any case, next at step 517 the foreground object of behavioral states, and detection of changes of behavior 

interest may be refined using edge information to more patterns in addition to individual behaviors. Development 
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activities have been complete to validate various scientific determined by analyzing the local variations of each pixel of 

definition of mouse behaviors and to create novel digital a series of frames over a short time period as indicated in 

video processing algorithms for mouse tracking and behav- step 509 of FIG. 5. The standard deviation from the mean is 

ior recognition, which are embody in software and hardware first calculated for each pixel. If the standard deviation is 
system according to the present invention. 5 greater than a chosen threshold, we tag those pixels as being 

Various lighting options for videotaping have been evalu- non-stationary or varying pixels. Those pixels that are below 

ated. Lighting at night as well as with night vision cameras the threshold may be tagged as stationary or constant pixels, 

was evaluated. It has been determined that good quality Only those stationary pixels are used in the averaging 

video was obtained with normal commercial video cameras process to calculate the background. Since the varying 

using dim red light, a frequency that is not visible to rodents, pixels are not used, there will be "holes" in each image that 

Videos were taken in a standard laboratory environment is being used in the averaging process. Over time, not all 

using commercially available cameras 105, for example a frames will have these holes at the same location and hence, 

Sony analog camera, to ensure that the computer algorithms ^ complete background image may be obtained with the 

developed would be applicable to the quality of video averaging process. Once the background image has been 
available in the average laboratory. The commercially avail- 15 obtained, subtraction of the background image from the 

able cameras with white lighting gave good results during given analyzed image yields the foreground objects. One 

the daytime and dim red lighting gave good results at night exemplary algorithm for such a background subtraction 

time. method will now be described. 

Referring again to FIG. 3, the first step in the analysis of T be the number of frames that are being averaged to 

home cage behavior is an automated initialization step that calculate the background. Let p^^^ ,^ be the pixel_value at 

involves analysis of video images to identify the location position (x,y) and frame number t, then the mean, p^^^^, and 

and outline of the mouse, as indicated by step 310. Second, standard deviation, o^^yy for that location are defined 

the location and outline of the mouse are tracked over time, respectively as, 
as indicated by step 315. Performing the initialization step 

periodically may be used to reset any propagation errors that i ^ 

appear during the tracking step. As the mouse is tracked over ^^'^^ " rZj ^^'•'•'^ 
time, its features including shape are extracted, and used for 

training and classifying the posture of the mouse from frame 1 ^ 2 

to frame, as indicated by step 320. Posture labels are = "fryZj ^^('•J'-'i-Pti.y)) 

generated for each frame, which are analyzed over time to ' * 
determine the actual behavior, as indicated by step 325. 

These steps will now be described in detail using the If the standard deviation, o^^y for a particular pixel is 

particular application of mouse behavior characterization. greater than a threshold, for example an intensity of 64 on 

T . c Ihe scale of 0 to 255 was used for a video clip with mouse 

I. Mouse Identification 35 . ♦u •* - j r i_ 1 j ■ 

in a cage, then it is omitted from the background image 

A typical video frame of a mouse in its home cage is calculation. 

shown in FIG. 6. In this video frame a mouse is shown in a Typically, the background image of a video session does 

rearing up posture. Many such frames make up the video of, not remain constant for a great length of time. For example, 

for example, a 24 hour mouse behavior monitoring session. in the case of monitoring mouse behavior the bedding in the 

As previously indicated, there are several approaches avail- mouse cage can shift due to the activity of the mouse. Hence, 

able for identifying and tracking moving objects in a scene. the background may need to be recalculated periodically. 

One of the simplest and most straightforward methods is External factors such as change in illumination conditions 

background subtraction of which one example was provided may require background image recalculations. If the camera 

in FIG. 5. 105 moves, then the background image might need to be 



A. Background Subtraction 



recalculated. 

Another method, other than performing background recal- 

Background subtraction as used in the present invention culations periodically, is to keep track of the difference 

generally involves generating a still background image from image and note any sudden changes such as an increase in 

all or a subset of the frames in a video cUp and subtracting 50 the number of white pixels in the difference image or the 

this background image from any given image to obtain the appearance of patches of while pixels in another area of the 

foreground objects. difference image. 

The background is generated by averaging many frames. An example of some screen shots of one exemplary 

for example approximately 100 frames of the video, after background subtraction process used for monitoring a 

compensating for any shifts caused by the motion of the 55 mouse with the present invention is shown in FIGS. 7A, 7B 

camera. Even if foreground objects are present in the frames and 7C. FIG. 7A illustrates a first frame in a sequence with 

that are being averaged to generate the background image, the mouse in an eating posture 705. FIG. 7B illustrates the 

their unwanted contribution is negligible when large num- same frame of the video image now having the area of the 

bers of frames are used for the background calculation, frame in which the pixels are changing identified as a 

assuming that the foreground object does not remain at the 60 blocked out 710. As a resuh the background has a "hole" 710 

same location throughout. Nevertheless, it may be helpful to (shown in black). This hole 710 will be filled with an image 

not consider those pixels where the foreground object is indicative of the true complete backgrouind image when 

present. other frames are averaged with it. 

In one implementation of the background averaging In order to generate a good background from a video 

process, only the stationary pixels in an image are consid- 65 sequence, several samples should first be generated. For 

ered to avoid the unwanted contributions of the foreground example, a 10-20 frame sample (30 frames per second) from 

moving objects. The stationary and non-stationary pixels are a video clip is taken and then averaged to generate one 
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sample. Once a sample is obtained, it may be used to update between foreground and background for the image in FIG. 

a previously existing background. A sufiBciently complete 7A. FIG. 8B illustrates the image after thresholding showing 

background may be obtained by averaging a number of the foreground mouse 705 object correctly identified. FIG. 

sample sets, for example 5-10 samples sets. FIG. 7C illus- 8C illustrates a video image showing the foreground object, 
irates the resulting background image for the video clip once 5 a mouse correctly identified with a polygon outline 805, 

the group of frames in a sample set and a number of sample created using convex hull approach as described above, 

sets are averaged together. As can be seen in FIG. 7C, this Another method of location and outline identification that 

method is quite successful at generating a reasonably com- may improve the representation of the shape of the mouse is 

plete background image (less the foreground object of the b-spline method. B-spline are piecewise polynomial 
interest) to be used in the background subtraction process for lO functions that can provide local approximation of contours 

identifying and segregating a desired object, in this case a of shapes using a small number of parameters and the 

inouse. piecewise smooth lines can be used to represent the outhne 

One primary advantage of this technique is its low com- of the object area. This is useful because human perception 

plexity that enables the background recalculations and fore- of shapes is deemed to be based on curvatures of parts of 

ground object detection to be performed with ease. This contours (or object surfaces). This is especially true since 

makes the background subtraction method of the present shapes of mice are curvatures at any time. This representa- 

invention well suited for use in real-time processing appli- tion may thus results in compression of boundary data as 

cations. well as smoothing of coarsely digitized contours. 

Suppose the mouse shape extracted is represented as a set 
B. Other Algorithms for Mouse Identification 20 of ordered boundary points W-(X,, Y,), with 0^i<n. This 

Various other algorithms may be used for object or mouse set of points is to be approximated by a B-sphne represen- 

identification. For example, one might use a mixture model tation as follows: 
and/or robust estimation algorithms in addition to, or in 
place of, background subtraction. These algorithms are 

newly developed theory in image sequence processing and f\t) = ^jOtBtit) 
object segmentation. They may handle object segmentation 
better than background subtraction in certain circumstances. 

Prehminary analysis indicates that mixture model and/or where the B^^ are modified B-spline basis functions, and 

robust estimation algorithms may have excellent results for P(0)=P(N) to constitute a close shape. Q,=(Qir» Q2/) are so 

mouse identification. called control points, which are not only the coefficients in 

this equation, but also physically define vertices of a poly- 

II. Location and Outline Identification and Feature gon that guides the spUnes to trace a smooth curve. Using 

Extraction standard Least-square minimization method, to minimize: 
In any case, once the background has been generated, it is 35 

then used to determine the foreground objects by taking the V v2 2 

intensity difference and applying a threshold detenmination ^ " ^ ^^^^^'^ " ^ ^''"'^''^ ~ 
procedure to remove noise. This step may involve threshold 
determination on both the intensity and the size of region. 

An 8-connection labeling procedure may be performed to where t,- is the knots associated with k where the spline 

screen out disconnected small noisy regions and improve the functions are tied together. Two equations can be obtained: 
region that corresponds to the mouse. In the labeling 

process, all pixels in a frame will be assigned a label as «-» --i 

foreground pixel or background pixel. Thresholding has " Zi^'f'^^^' 
generated labels for certain pixels. Neighbors of those 

labeled pixels that have not been labeled may obtain the "-1 1-1 

same label as the labeled pixel. Eight-connectedness defines X ^'-^ = ^ ^i(o)>i 

8 corner-adjacent pixels that are all neighbors. The remain- 

ing regions indicated to be foreground objects are much 

smaller compared to the region of mouse, thus a size criteria where 

is used to select the larger mouse region. The outline or 

contour of this foreground object is thus determined. „_i 

Further, the convex hull of the pixels is used in the 0 s / <w and m,^ = 25,(ri)fl,(ft). 

foreground object for representation. Convex hull H of an *^ 
arbitrary set S, which is a region in the frame in this case, is 55 

the smallest set containing S. The set difference H-S is called Based on the silhouette obtained from background 

the convex deficiency D of the set S. The region S' boundary subtraction, a series of image processing procedures may be 

can be partitioned by following the contour of S and marking performed first to detect edge using a sobel edge detection 

the points at which transition is made into or out of a algorithm and then, using morphological operations to trim 
component of the convex deficiency. These marking points 60 edge points to ensure that the edge points are singly chained, 

can be connected into a polygon that gives a description of in order to proceed, a fixed reference point on the closed 

the region. The centroid (or center of mass) of the fore- shape is required. We make use of the features we have 

ground object is calculated and is used for representing the extraaed for the shape and use the angle, which indicates the 

location of the object (e.g.. mouse). direction of the principal axis derived through Principle 
FIGS. 8 A, 8B and 8C illustrate the results of the location 65 Component Analysis (CPA), to derive that reference point, 

and object outhne identification for a mouse using the A straight line that goes through the centroid with that angle 

present invention. FIG. 8 A illustrates a difference image is generated and the point at which this straight line inter- 
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sects the edge pixel is the reference point. We use the Using the central moments, we define, 
reference point as the starting point to order those edge 

pixels clockwise to facilitate the solution of the above i 2mi,i 

equations. We obtain a matrix of the control points, which ^ ^arctan^^^^ _ 

define uniquely the b-spUne function. 5 m m + W 

One example of the use of the B-Spline algorithm imple- E = "^^^ ^ '"^■^ 



mcnted in the present invention is illustrated in FIGS. ('m:,© +^0,2)^ 
9A-9C. The original image, the detected edge, and the 

plotted b-spline function where the vertices of the curve are r is equal to the ratio of the length of the range of the points 

the control points, are shown in RGS. 9A, 9B, and 9C, 10 projected along an axis perpendicular to 0, to the length of 

respectively. FIG. 9 A illustrates an exemplary video image the range of the points projected along an axis parallel to 0. 

frame of mouse eating 705. FIG. 9B illustrates the seg- This may also be defined as the aspea ratio (ratio of width 

mented edge 905 of the mouse 705 found in FIG. 9A. FIG. to length) after rotating the foreground object by 0. 

9C illustrates a b-spline representation of the mouse edge Tracking in the feature space involves following feature 

910 extrapolated from the segmented edge of the mouse i5 values from one frame to the next. For example, if the area 

found in FIG. 9 A. steadily increases, it could mean that the mouse is coming 

As a result, either b-spline representation, or convex hull out of a cuddled up position to a more elongated position, or 

representation can be used as features of foreground object, that it could be moving fi-om a fi-ont view to a side view, etc. 

in addition to other features that include but not limited to: if the position of the centroid of the mouse moves up, it 

centroid, the principal orientation angle of the object, the 20 means that the mouse may be rearing up on its hind legs, 

area (number of pixels), the eccentricity (roundness), and the Similarly, if the angle of orientation changes from horizontal 

aspect ratio of object. to vertical, it may be rearing up. These changes can be 

analyzed with combinations of features ako, 

III. Mouse tracking However, it is possible for a b-spline representation to be 

Ideal tracking of foreground objects in the image domain "^^^ near-optimal tracking efficiently in the 

involves a matching operation to be performed that identifies l^^^^e domain (i.e., the complete unage before background 

corresponding points from one frame to the next. This ^ 

process may become computationally too consuming or ,,7 t> . 

expensive to perform in an efficient manner. Thus, one 30 ^"""^ Classification 

approach is to use approximations to the ideal case that can Once the features are obtained for the frames in the video 

be accomplished in a short amount of time. For example, sequence, the foreground state of the mouse is classified into 

tracking the foreground object may be achieved by merely one of the given classes. This involves building a classifier 

tracking the outline contour from one frame to the next in the that can classify the shape using the available features. This 

feature space (i.e., identified foreground object image). 35 information may be stored in, for example, a database in, for 

Id one variation of the invention, tracking is performed in example, a data memory. In one variation of the invention a 

the feature space, which provides a close approximation to Decision Tree classifier (e.g., object shape and posture 

tracking in the image domain. The features include the classifier 215) was implemented by training the classifier 

centroid, principal orientation angle of the object, area with 488 samples of digitized video of a standard, in this 

(number of pixels), eccentricity (roundness), and the aspect 40 case, normal mouse. Six attributes (or features) for each 

ratio of object with lengths measured along the secondary sample were identified. Five posture classes for classifica- 

and primary axes of the object. In this case, let S be the set tion were identified as listed below, 

of pixels in the foreground object, A denote the area in 1. Horizontally positioned, side view, either in normal 

number of pixels, (C^ Cy) denote the centroid, 0 denote the state or elongated. 

orientation angle, E denote the eccentricity, and R denote the 45 ^ ^^^.^^^ positioned, either rearing or hanging from 

aspect ratio. ^^^ ^ Pjq3 ^ 

Then. 3. Cuddled up position (like a ball). 

4. Horizontally positioned, but either front or back view, 
50 i.e., axis of mouse along the viewer's line of sight. 

5. Partially reared, e.g., when drinking or eating, sitting on 
hind legs (e.g., See FIG. 7A). 

The system of the present invention was exercised using 
these classifications. The distribution of the samples 
55 amongst the five classes is shown in Table 1. Performing a 
Let us define three intermediate terms, called second order 10-fold cross-validation on the 488 training samples, a 
moments combined accuracy of 93.65% was obtained indicating that 

the classifier was performing well. This in the range of the 
highest levels of agreement between human observers. The 
^ (x-c^r cross-validation procedure involves randomly dividing a 

training set into N approximately equal sets, and for each of 
'"0.2 = ^ (>• - Cyf the N folds or iterations, one set is set aside for testing while 

^ the remaining N-1 sets arc used as training samples. Accu- 

mi.i -^U-c,)(>-Cj,) racy values for individual classes are indicated in the last 

s 65 column of Table 1, Table 2 shows the overall accuracy 

values for each fold. We assign appropriate labels for each 
firame depending on the class to with it was classified to. 
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TABLE 1 




Difitribudon of samptes in the five classes 




and the accuracy values for < 


:ach dass. 


Class 


Number of Samples 


Accuracy (%) 


] 


109 


104/109 - 95.41 


2 


103 


103/103 = 100.0 


3 


106 


92/106 « 86.79 


4 


82 


75/82 = 91.46 


5 


88 


83/88 = 94.32 


Total 


488 


457/488 = 93.65 



It is evident from the data in the Table 1 that class 2 was 
arguably the most easily for the automated system to iden- 
tify. This is because the vertical position is quite distinct 
from the other postures. On the other hand, classes 3 and 4 
yield the poorest results due to similarity in the two postures. 
Both classes depict the mouse as a fairly round object, the 
only primary difference being the size of the object — class 
3 sizes are expected to be slightly larger than those from 
class 4. 

TABLE 2 



Accuracy results for each fold 
for a cross-validation test. 



Fold Accuracy 



1 


91.67 


2 


89.58 


3 


93.75 


4 


91.67 


5 


95.83 


6 


95.83 


7 


93.75 


8 


97.92 


9 


95.83 


10 


91.07 


Overall 


93.65 



As illustrated by the tables, the present system provides 
good accuracy for mouse shape and posture recognition and 
classification. 

V. Behavior Identification 

Once the postures in all the frames in the video clip have 
been labeled, we now need to determine certain pre-defined 
behaviors as defined in the database based on the postures 
that have been identified. Currently, 23 such behaviors have 
been defined which include: sleep, groom, eat, rear up on the 
hind legs, drink, walk, jump, hang from the top of the cage, 
stretch, dig, awaken, arousal, twitch, stretch, yawn, pause, 
circle, forage, chew, urinate, defecate. In addition, grooming 
is divided into licking and scratching and rearing up into 
supported and unsupported. 

This process will be accomplished in real-time so that 
immediate results will be reported in investigators or stored 
in a database. One approach is to use a rule-based label 
analysis procedure (or a token parsing procedure) by which 
the sequence of labels is analyzed and identify particular 
behaviors when its corresponding sequence of labels is 
derived from a video frame being analyzed. For example, if 
a long sequence (lasting for example several minutes) of the 
"Cuddled up position" label (Class 3) is observed, and if its 
centroid remains stationary, then, it may be concluded that 
the mouse is sleeping. If the location of the waterspout is 
identified, and if we observe a series of "partially reared" 
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(Class 5) labels, and if the position of the centroid, and the 
mouse's angle of orientation fall within a small range that 
has been predetermined, the system can determine and 
identify that the mouse is drinking. It may also be useful for 
^ certain extra conditions to be tested such as, "some part (the 
mouth) of the mouse must touch the spout if drinking is to 
be identified" in addition to temporal characteristics of the 
behavior. 

10 Another approach involves using a probabilistic model 
such as Hidden Markov Models (HMMs), where models 
may be built for each class of behavior with training 
samples. These models may then be used to identify behav- 
iors based on the incoming sequence of labels. The HMM 

^5 can provide significant added accuracy to temporal relation- 
ships for proper complex behavior characterization. 

Referring now to FIG. 10, various exemplary mouse state 
transitions tested in the present invention are illustrated. The 

20 five exemplary mouse state transitions include: (1) Horizon- 
tal (HS) 1005, (2) Cuddled up (CU) 1010, (3) Partially 
reared (PR) 1015, (4) Vertically Reared (VR) 1020, and (5) 
Forward Back (FB) 1025 postures. As illustrated, FIG. 10 
shows the five posture states and the duration for which a 

25 mouse spent in each state in an exemplary sample video clip. 
One example of a pattern that is understandable and evident 
from the figure is that the mouse usually passes through the 
partially reared (PR) 1015 state to reach the vertically reared 
(VR) 1025 state from the other three ground-level states. 
The states are defined according to the five posture classes 
mentioned previously. 

Many important features can be derived from this 
representation, e.g., if the state changes are very frequent, it 
35 would imply that the mouse is very aaive. If the mouse 
remained in a single ground -level state such as "cuddled-up" 
(class 3) for an extended period of time, the system may 
conclude that the mouse is sleeping or resting. The sequence 
of transitions arc also important, e.g., if the mouse rears 
(class 2) from a ground-level state such as "Horizontally 
positioned" (class 1), it should pass briefly through the 
partially reared state (class 5). Techniques such as HMMs 
exploit these types of time-sequence-dependent information 
for performing classification. 

A simple HMM system has been created using dynamic 
programming to find the best match between the input 
sequence and paths through the state machine. It has been 
used to classify events in one of the mouse behavior 
50 sequences. The HMM system was provided with a sequence 
of tokens representing recognized actions or views from a 
benchmark mouse-rear video; this file includes views from 
five different postures, which are: 

c=cuddled_posture view 
/=front_or__badL_view 
/i»horizontal_side_view 

60 

/?=paitially_rearcd_view 

rtareaied_Qr_vertical_view (i) 

65 Each of these represents a posture of the mouse and all 
together they constitute five (5) tokens. These tokens cause 
the HMM to go from one (hidden) state to another. The 
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HMM may classify behavior into one of, for example, four 
hidden states: horizontal, rearing, cuddled, or indecisive: 



0 o horizoDta]; 1 «» cuddled; 2 « rearing; 3 = indecisive; (2) 



Thus, the 
described as: 



HMM defining mouse behaviors can be 



size 4 5; 



start 0 stop 0; 



symbols c f h p r 
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This approach to a HMM for mouse behavior character- 
ization may result in a number of mismatched cases which 
may be categorized into three (3) types: (a) one mismatch 
(the last token) because the start and stop states were forced 
to be 0; (b) the PARTIALLY_REARED may be mapped to 
indecisive, but this may only be a difference in the naming; 
and (c) the FRONT„OR_BACK may be mapped to the 
same value as HORIZ_SIDE_VIEW (21 cases). However, 
it is reasonable that both FRONT_OR_BACK and 
H0RIZ_S1DE_VIEW are mapped to the same classifica- 
tion because both are similar to each other "behaviorally", 
i.e., from the mouse *s point of view, being FRONT_OR__ 
BACK or HORIZ__SIDE are the same thing. This may yield 
a perfect mapping from input to output. This is but one 
exemplary approach for the frame work for defining a HMM 
analysis for determining mouse behavior. 

Although the above exemplary embodiment is directed to 
a mouse analyzed in a home cage, it is to be understood that 
the mouse (or any object) may be analyzed in any location 
or environment. Further, the invention in one variation may 
be used to automatically detect and characterize one or more 
particular behaviors. For example, the system could be 
configuired to automatically detect and characterize an ani- 
mal freezing and/or touching or sniffing a particular object. 
Also, the system could be configured to compare the object's 
behavior against a "norm" for a particular behavioral param- 
eter. Other detailed activities such as skilled reaching and 
forelimb movements as well as social behavior among 
groups of animals can also be detected and characterized. 

In summary, when a new video clip is analyzed, the 
system of the present invention first obtains the video image 
background and uses it to identify the foreground objects. 
Then, features are extracted from the foreground objects, 
which are in turn passed to the decision tree classifier for 
classification and labeling. This labeled sequence is passed 
to a behavior identification system module that identifies the 
final set of behaviors for the video chp. The image resolution 
of the system that has been obtained and the accuracy of 50 
identification of the behaviors attempted so far have been 
very good and resulted in an effective automated video 
image object recognition and behavior characterization sys- 
tem. 

The invention may identify some abnormal behavior by 
using video image information (for example, stored in 
memory) of known abnormal animals to build a video 
profile for that behavior. For example, video image of 
vertical spinning while hanging from the cage lop was stored 
to memory and used to automatically identify such activity 
in mice. Further, abnormalities may also result from an 
increase in any particular type of normal behavior. Detection 
of such new abnormal behaviors may be achieved by the 
present invention detecting, for example, segments of 
behavior that do not fit the standard profile. The standard 
profile may be developed for a particular strain of mouse 
whereas detection of abnormal amounts of a normal behav- 
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ior can be detected by comparison to the statistical proper- 
ties of the standard profile. Thus, the automated analysis of 
the present invention may be used to build a profile of the 
behaviors, their amount, duration, and daily cycle for each 
animal, for example each commonly used strain of mice. A 
plurality of such profiles may be stored in, for example, a 
database in a data memory of the computer. One or more of 
these profile may then be compared to a mouse in question 
and difference from the profile expressed quantitatively. 

The techniques developed with the present invention for 
automation of the categorization and quantification of all 
home-cage of mouse behaviors throughout the daily cycle is 
a powerful tool for detecting phenotypic effects of gene 
manipulations in mice. As previously discussed, this tech- 
nology is extendable to other behavior studies of animals 
and humans, as well as surveillance purposes. In any case, 
the present invention has proven to be a significant achieve- 
ment in creating an automated system and methods for 
automated accurate identification, tracking and behavior 
categorization of an object whose image is captured in a 
video image. 

Although particular embodiments of the present invention 
have been shown and described, it will be understood that it 
is not intended to limit the invention to the preferred or 
disclosed embodiments, and it will be obvious to those 
skilled in the art that various changes and modifications may 
be made without departing from the spirit and scope of the 
present invention. Thus, the invention is intended to cover 
alternatives, modifications, and equivalents, which may be 
included within the spirit and scope of the invention as 
defined by the claims. 

For example, the present invention may also include 
audio analysis and/or multiple camera analysis. The video 
image analysis may be augmented with audio analysis since 
audio is typically included with most video systems today. 
As such, audio may be an additional variable used to 
determine and classify a particular objects behavior. Further, 
in another variation, the analysis may be expanded to video 
image analysis of multiple objects, for example mice, and 
their social interaction with one another. In a still further 
variation, the system may include multiple cameras provid- 
ing one or more planes of view of an object to be analyzed. 
In an even further variation, the camera may be located in 
remote locations and the video images sent via the Internet 
for analysis by a server at another site. In fact, the standard 
object behavior data and/or database may be housed in a 
remote location and the data files may be downloaded to a 
stand alone analysis system via the Internet, in accordance 
with the present invention. These additional features/ 
functions adds versatility to the present invention and may 
improve the behavior characterization capabilities of the 
present invention to thereby achieve object behavior catego- 
rization which is nearly perfect to that of a human observer 
for a broad spectrum of applications. 

All publications, patents, and patent applications cited 
herein are hereby incorporated by reference in their entirety 
for all purposes. 

What is claimed is: 

1. A system comprising: 

a computer configured to determine a position and shape 
of an object of interest from video images and to 
characterize activity of said object of interest based on 
analysis of changes in said position and said shape over 
time; 

wherein said computer includes an object identification 
and segregation module receiving said video images; 
and 
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wherein said object identification and segregation module 
operates using a background subtraction algorithm in 
which a plurality of said video images are grouped into 
a set, a standard deviation map of the set of video 
images is created, a bounding box where a variation is 5 
greater than a predetermined threshold is remove from 
said set of video images, and the set of images less said 
bounding boxes is averaged to produce a back ground 
image. 

2. The system of claim 1, further comprising: lo 
a video camera coupled to said computer for providing 

said video images. 

3. The system of claim 2, further comprising: 

a video digitization unit couple to said video camera and 
said computer for converting said video images pro- 
vided by said video camera from analog to digital 
format. 

4. The system of claim 3, further comprising: 

a storage/retrieval unit coupled to said video digitization 
unit, said video camera, and said computer, for storing 
video images and standard object video images. 

5. The system of claim 1, wherein said computer further 
includes a behavior identification module for characterizing 
activity of said object, said behavior identification module 
being coupled to said object identification and segregation 
module. 

6. The system of claim 5, wherein said computer further 
includes an object tracking module for tracking said object 
from one frame of said video images to another frame, and 
an object shape and location change classifier for classifying 
the activity of said object, coupled to each other, said object 
identification and segregation module, and said behavior 
identification module. 

7. The system of claim 6, wherein said computer further 
includes a standard object behavior storage module that 
stores information about known behavior of a predetermined 
standard object for comparing the activity of said object, 
said standard object behavior storage module being coupled 
to said behavior identification module, and a standard object 
classifier module coupled to said standard object behavior 
module. 

8. The system of claim 1, wherein said computer further 
includes a standard object behavior storage module that 
stores information about known behavior of a predetermined 
standard object for comparing the activity of said object, 
said standard object behavior storage module being coupled 
to said behavior identification module. 

9. The system of claim 1, wherein said object is a Uving 
object. 

10. The system of claim 1, wherein said object is an 
animal. 

11. The system of claim 1, wherein said object is a mouse. 

12. The system of claim 1, wherein said object is a human. 

13. Ttic system of daim 1, wherein said object is a man 
made machine. 

14. A method of characterizing activity of an object using 
a computer comprising: 

detecting a foreground object of interest in video images; 
tracking said foreground object over a plurality of said 
video images; 

classifying said foreground object in said plurality of 

video images; and 
characterizing said activity of said foreground object 

based on comparison of said classifications to activity 65 

of a standard object; wherein said characterizing said 

activity includes: 
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describing a sequence of postures as behavior primi- 
tives; and 

aggregating behavior primitives into actual behavior 
over a range of images; 

wherein said describing said behavior primitives fur- 
ther includes: 

identifying patterns of postures over a sequence of 
images; and 

analyzing temporal information selected from the 
group consisting of direction and magnitude of 
movement of the centroid, increase and decrease 
of the eccentricity, increase and decrease of the 
area, increase and decrease of the aspect ratio of 
the bounding box, change in the b -spline repre- 
sentation points, change in the convex hull points, 
and direction and magnitude of corner points. 

15. The method of claim 14, wherein said foreground 
object detection includes the step of generating a back- 
ground image from an average of a set of individual frames 
of said video images. 

16. The method of claim 15, wherein said step of gener- 
ating a background image includes the step of determining 
variation in intensity of pixels within said individual frames 
to identify a region where said foreground object is located. 

17. The method of claim 16, wherein said step of gener- 
ating a background image further includes the step of using 
non-variant pixels of the video images to generate said 
background image. 

18. The method of claim 17, wherein said step of gener- 
ating a background image is performed periodically to 
correct for changes in background objects and small move- 
ments of a camera capturing said video images. 

19. The method of claim 14, wherein said detecting a 
foreground object includes using a background subtraction 
method comprising the steps of: 

multiply frames in a neighborhood of current image; 

apply a lenient threshold on a difference between a current 
image and a background so as to determine a broad 
region of interest; 

classify by intensity various pixels in said region of 
interest to obtain said foreground object; and 

apply edge information to refine contours of said fore- 
ground object image. 

20. The method of claim 14, wherein said step of detect- 
ing said foreground includes the step of manual identifica- 
tion of foreground objects to be tracked and characterized. 

21. The method of claim 14, wherein said posture deter- 
mination and description includes using statistical and 
contour-based shape information. 

22. The method of claim 21, wherein said step of iden- 
tifying and classifying changes to said foreground object 
includes using statistical shape information selected from 
the group consisting of: 

area of the foreground object; 
centroid of the foreground object; 
bounding box and its aspect ratio of the foreground 
object; 

eccentricity of the foreground object; and 

a directional orientation of the foreground object relative 

to an axis as generated with a Principal Component 

Analysis. 

23. The method of claim 21, wherein said step of iden- 
tifying and classifying changes to said foreground object 
uses contour-based shape information selected from the 
group consisting of b-spline representation, convex hull 
representation, and comer points. 
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24. The method of claim 21, wherein said step of iden- 
tifying and classifying changes to said foreground object 
includes identifying a set of model postures and their 
description information, said set of model postures including 
horizontal posture, vertical posture, eating posture, or sleep- 5 
ing posture. 

25. The method of claim 24, wherein said step of iden- 
tifying and classifying changes to said foreground object 
includes classifying changes to said foreground object 
includes classifying the statistical and contour-based shape lo 
information from a current image to assign a best-matched 
posture. 

26. The method of claim 25, wherein the said step of 
determining actual behavior by aggregating behavior primi- 
tives includes the step of analyzing temporal ordering of the 15 
primitives, such as using information about a transition from 

a previous behavior primitive to a next behavior primitive. 

27. The method of claim 26, wherein said temporal 
analysis is a time-series analysis such as Hidden Markov 
Model (HMMs). 20 

28. The method of claim 26, wherein the said step of 
determining actual behavior includes identifying actual 
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behavior selected from a group consisting of sleeping, 
eating, roaming around, grooming, and climbing. 

29. A method for background subtraction of a video 
image, comprising the steps of: 

grouping a number of images into a set of video images; 

creating a standard deviation map of the grouped images; 

removing a bounding box area of said image where 
variation is above a predetermined threshold to create 
a/partial image; and 

combining said partial image wiih,an existing set of 
partial images by averaging the set of images to gen- 
erate a complete background image deplete of a desired 
foreground object. 

30. The method of claim 29, further comprising the step 
of subtracting said complete background image from a 
current image so a to obtain said desired foreground object. 

31. The method of claim 29, wherein said steps are 
repeated periodically to update said complete background 
package. 
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